feat: add wait command with non-zero exit on failure by 183amir · Pull Request #28 · idiap/gridtk

183amir · 2026-03-16T16:10:46Z

Summary

Add gridtk wait command that polls jobs until all reach a terminal state
Progress shows state breakdown: Waiting for 5 job(s): 3 pending, 2 running
Exits with code 1 if any job ended in a failed state (FAILED, CANCELLED, TIMEOUT, etc.)
Supports --interval option and all standard job filters
Fix JobManager.__del__ deleting DB due to SQLite session isolation — moved to explicit cleanup_empty_database() called after every CLI command
Document wait command in README

Examples

$ gridtk wait
Waiting for 1 job(s): 1 running (checking every 10s)
Job 1: COMPLETED (0)

$ echo $?
0

$ gridtk wait -j 1 --interval 30
Job 1: FAILED (1)

$ echo $?
1

🤖 Generated with Claude Code

📚 Documentation preview 📚: https://gridtk--28.org.readthedocs.build/en/28/

github-actions · 2026-03-16T16:39:42Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/gridtk
cli.py					690-691, 704-709, 715
manager.py
Project Total

_{This report was generated by python-coverage-comment-action}

src/gridtk/cli.py

The base branch was changed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace ctx.exit(1) with raise SystemExit(1) to ensure click.echo output is flushed before the process exits. ctx.exit(1) can suppress output in some Python/Click versions (e.g. Python 3.9). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Ensures the DB session is properly committed before raising SystemExit to avoid leaving the database in a bad state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Use return_value instead of side_effect iterator to avoid mock exhaustion from GC-triggered __del__ calls on some Python versions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Prevents __del__ race condition on CI where GC runs between submit and subsequent commands, deleting the DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

__del__ was deleting the DB on GC due to SQLite session isolation — a new session in __del__ couldn't see rows committed by another engine instance. Now __del__ only disposes the engine, and DB cleanup is called explicitly after delete commands. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

183amir · 2026-03-17T12:28:00Z

@Yannick-Dayer Changes since your last review:

Fixed __del__ bug: JobManager.__del__ was deleting the DB due to SQLite session isolation (new session in __del__ couldn't see rows committed by another engine instance). Moved cleanup to an explicit cleanup_empty_database() method called after every CLI command via process_result. This deletion is needed for e.g. where gridtk list in a random folder would leave behind an empty jobs.sql3.

All tests pass, tested on real Slurm 20.11.4. Ready for re-review.

Yannick-Dayer

Good catch on that __del__ bug.

The following comment is a suggestion and not required if you don't feel like applying it, so I'm approving again.

Yannick-Dayer · 2026-03-17T13:41:50Z

src/gridtk/manager.py

@@ -312,4 +312,6 @@ def __del__(self):
                Path(self.database).unlink()
                if self.logs_dir.exists() and len(os.listdir(self.logs_dir)) == 0:


You could use pathlib's Path.rglob instead of os.listdir

Suggested change

if self.logs_dir.exists() and len(os.listdir(self.logs_dir)) == 0:

if self.logs_dir.exists() and len(list(self.logs_dir.rglob("*"))) == 0:

Or even Path.iterdir() as suggested by the pathlib documentation

sorry I missed this. I will keep it in mind for the next iteration. Thank you for all the quick review feedbacks. I am using gridtk with agents and they surprisingly use it really well without introduction.

183amir requested a review from Yannick-Dayer March 16, 2026 16:56

183amir changed the base branch from main to feat/json-output March 16, 2026 17:02

183amir force-pushed the feat/wait-command branch from bfb8ffc to d0c3beb Compare March 16, 2026 17:02

183amir mentioned this pull request Mar 16, 2026

chore(release): 3.2.0 #31

Closed

Yannick-Dayer reviewed Mar 16, 2026

View reviewed changes

src/gridtk/cli.py Outdated Show resolved Hide resolved

183amir force-pushed the feat/json-output branch from 46fd3ff to b225b2b Compare March 17, 2026 08:47

183amir force-pushed the feat/wait-command branch from 7185868 to fd30aad Compare March 17, 2026 08:47

Yannick-Dayer previously approved these changes Mar 17, 2026

View reviewed changes

Base automatically changed from feat/json-output to main March 17, 2026 09:02

Yannick-Dayer previously approved these changes Mar 17, 2026

View reviewed changes

183amir dismissed Yannick-Dayer’s stale review via 5110b54 March 17, 2026 09:04

Amir Mohammadi and others added 5 commits March 17, 2026 11:43

feat: add wait command with non-zero exit on failure

2f41009

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: commit session before SystemExit in wait command

40556e3

Ensures the DB session is properly committed before raising SystemExit to avoid leaving the database in a bad state. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

docs: document wait command in README

5e04599

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: show state breakdown in wait progress (pending vs running)

44f8e71

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

183amir force-pushed the feat/wait-command branch from 5110b54 to 44f8e71 Compare March 17, 2026 10:44

183amir enabled auto-merge (squash) March 17, 2026 10:46

Amir Mohammadi and others added 7 commits March 17, 2026 13:10

fix: robust wait test and formatting

25dca2b

Use return_value instead of side_effect iterator to avoid mock exhaustion from GC-triggered __del__ calls on some Python versions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: force GC in wait test to prevent __del__ race condition

5c41ce4

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

debug: add list assertion before wait

1446a95

debug: assert DB exists after submit

6952b82

fix: add gc.collect() to all tests that submit then query

d8adb6b

Prevents __del__ race condition on CI where GC runs between submit and subsequent commands, deleting the DB. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

debug: trace __del__ and DB existence on CI

f4a5d8c

183amir force-pushed the feat/wait-command branch from cc2d024 to a475127 Compare March 17, 2026 12:26

Yannick-Dayer approved these changes Mar 17, 2026

View reviewed changes

183amir merged commit a97666f into main Mar 17, 2026
10 checks passed

183amir deleted the feat/wait-command branch March 17, 2026 13:44

183amir mentioned this pull request Mar 17, 2026

docs: recommend uv for installation in README #33

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add wait command with non-zero exit on failure#28

feat: add wait command with non-zero exit on failure#28
183amir merged 12 commits intomainfrom
feat/wait-command

183amir commented Mar 16, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

183amir commented Mar 17, 2026 •

edited

Loading

Uh oh!

Yannick-Dayer left a comment

Uh oh!

Yannick-Dayer Mar 17, 2026

Uh oh!

Yannick-Dayer Mar 17, 2026

Uh oh!

183amir Mar 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -312,4 +312,6 @@ def __del__(self):
		Path(self.database).unlink()
		if self.logs_dir.exists() and len(os.listdir(self.logs_dir)) == 0:

	if self.logs_dir.exists() and len(os.listdir(self.logs_dir)) == 0:
	if self.logs_dir.exists() and len(list(self.logs_dir.rglob("*"))) == 0:

Conversation

183amir commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Examples

Uh oh!

github-actions bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report

Uh oh!

Uh oh!

183amir commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Yannick-Dayer left a comment

Choose a reason for hiding this comment

Uh oh!

Yannick-Dayer Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Yannick-Dayer Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

183amir Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

183amir commented Mar 16, 2026 •

edited

Loading

github-actions bot commented Mar 16, 2026 •

edited

Loading

183amir commented Mar 17, 2026 •

edited

Loading